{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Training\n",
    "\n",
    "In this part you learn how to use MLRun's **Feature Store** to easily define a **Feature Vector** and create the dataset you need to run the training process.  \n",
    "By the end of this tutorial you’ll learn how to:\n",
    "- Combine multiple data sources to a single feature vector\n",
    "- Create training dataset\n",
    "- Create a model using an MLRun hub function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_name = \"fraud-demo\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:43:21,980 [info] loaded project fraud-demo from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "import mlrun\n",
    "\n",
    "# Initialize the MLRun project object\n",
    "project = mlrun.get_or_create_project(project_name, context=\"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Create a feature vector  \n",
    "In this section you create a feature vector.  \n",
    "The Feature vector has a `name` so you can reference to it later via the URI or your serving function, and it has a list of \n",
    "`features` from the available feature sets.  You can add a feature from a feature set by adding `<FeatureSet>.<Feature>` to \n",
    "the list, or add `<FeatureSet>.*` to add all the feature set's available features.  \n",
    "\n",
    "By default, the first FeatureSet in the feature list acts as the spine, meaning that all the other features are joined to it.  \n",
    "For example, in this instance you use the early sense sensor data as the spine, so for each early sense event you create produces a row in the resulted feature vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the list of features to use\n",
    "features = [\n",
    "    \"events.*\",\n",
    "    \"transactions.amount_max_2h\",\n",
    "    \"transactions.amount_sum_2h\",\n",
    "    \"transactions.amount_count_2h\",\n",
    "    \"transactions.amount_avg_2h\",\n",
    "    \"transactions.amount_max_12h\",\n",
    "    \"transactions.amount_sum_12h\",\n",
    "    \"transactions.amount_count_12h\",\n",
    "    \"transactions.amount_avg_12h\",\n",
    "    \"transactions.amount_max_24h\",\n",
    "    \"transactions.amount_sum_24h\",\n",
    "    \"transactions.amount_count_24h\",\n",
    "    \"transactions.amount_avg_24h\",\n",
    "    \"transactions.es_transportation_sum_14d\",\n",
    "    \"transactions.es_health_sum_14d\",\n",
    "    \"transactions.es_otherservices_sum_14d\",\n",
    "    \"transactions.es_food_sum_14d\",\n",
    "    \"transactions.es_hotelservices_sum_14d\",\n",
    "    \"transactions.es_barsandrestaurants_sum_14d\",\n",
    "    \"transactions.es_tech_sum_14d\",\n",
    "    \"transactions.es_sportsandtoys_sum_14d\",\n",
    "    \"transactions.es_wellnessandbeauty_sum_14d\",\n",
    "    \"transactions.es_hyper_sum_14d\",\n",
    "    \"transactions.es_fashion_sum_14d\",\n",
    "    \"transactions.es_home_sum_14d\",\n",
    "    \"transactions.es_travel_sum_14d\",\n",
    "    \"transactions.es_leisure_sum_14d\",\n",
    "    \"transactions.gender_F\",\n",
    "    \"transactions.gender_M\",\n",
    "    \"transactions.step\",\n",
    "    \"transactions.amount\",\n",
    "    \"transactions.timestamp_hour\",\n",
    "    \"transactions.timestamp_day_of_week\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import MLRun's Feature Store\n",
    "import mlrun.feature_store as fstore\n",
    "\n",
    "# Define the feature vector name for future reference\n",
    "fv_name = \"transactions-fraud\"\n",
    "\n",
    "# Define the feature vector using the feature store (fstore)\n",
    "transactions_fv = fstore.FeatureVector(\n",
    "    fv_name,\n",
    "    features,\n",
    "    label_feature=\"labels.label\",\n",
    "    description=\"Predicting a fraudulent transaction\",\n",
    ")\n",
    "\n",
    "# Save the feature vector in the feature store\n",
    "transactions_fv.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Preview the feature vector data\n",
    "\n",
    "Obtain the values of the features in the feature vector, to ensure the data appears as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:43:23,376 [info] wrote target: {'name': 'parquet', 'kind': 'parquet', 'path': 'v3io:///projects/fraud-demo-dani/FeatureStore/transactions-fraud/parquet/vectors/transactions-fraud-latest.parquet', 'status': 'ready', 'updated': '2023-02-15T14:43:23.375968+00:00', 'size': 140838, 'partitioned': True}\n"
     ]
    }
   ],
   "source": [
    "# Import the Parquet Target so you can directly save your dataset as a file\n",
    "from mlrun.datastore.targets import ParquetTarget\n",
    "\n",
    "# Get offline feature vector as dataframe and save the dataset to parquet\n",
    "train_dataset = fstore.get_offline_features(fv_name, target=ParquetTarget())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>event_details_change</th>\n",
       "      <th>event_login</th>\n",
       "      <th>event_password_change</th>\n",
       "      <th>amount_max_2h</th>\n",
       "      <th>amount_sum_2h</th>\n",
       "      <th>amount_count_2h</th>\n",
       "      <th>amount_avg_2h</th>\n",
       "      <th>amount_max_12h</th>\n",
       "      <th>amount_sum_12h</th>\n",
       "      <th>amount_count_12h</th>\n",
       "      <th>...</th>\n",
       "      <th>es_home_sum_14d</th>\n",
       "      <th>es_travel_sum_14d</th>\n",
       "      <th>es_leisure_sum_14d</th>\n",
       "      <th>gender_F</th>\n",
       "      <th>gender_M</th>\n",
       "      <th>step</th>\n",
       "      <th>amount</th>\n",
       "      <th>timestamp_hour</th>\n",
       "      <th>timestamp_day_of_week</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1763</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>45.28</td>\n",
       "      <td>144.56</td>\n",
       "      <td>5.0</td>\n",
       "      <td>28.9120</td>\n",
       "      <td>161.75</td>\n",
       "      <td>1017.80</td>\n",
       "      <td>33.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>24.02</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1764</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.81</td>\n",
       "      <td>47.75</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23.8750</td>\n",
       "      <td>68.16</td>\n",
       "      <td>653.02</td>\n",
       "      <td>24.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>26.81</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1765</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>33.10</td>\n",
       "      <td>91.11</td>\n",
       "      <td>4.0</td>\n",
       "      <td>22.7775</td>\n",
       "      <td>121.96</td>\n",
       "      <td>1001.32</td>\n",
       "      <td>32.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>14.95</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1766</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>22.35</td>\n",
       "      <td>37.68</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12.5600</td>\n",
       "      <td>71.63</td>\n",
       "      <td>1052.44</td>\n",
       "      <td>37.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>13.62</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1767</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>44.37</td>\n",
       "      <td>76.87</td>\n",
       "      <td>4.0</td>\n",
       "      <td>19.2175</td>\n",
       "      <td>159.32</td>\n",
       "      <td>1189.73</td>\n",
       "      <td>39.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>12.82</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      event_details_change  event_login  event_password_change  amount_max_2h  \\\n",
       "1763                     0            0                      1          45.28   \n",
       "1764                     1            0                      0          26.81   \n",
       "1765                     0            1                      0          33.10   \n",
       "1766                     0            0                      1          22.35   \n",
       "1767                     0            0                      1          44.37   \n",
       "\n",
       "      amount_sum_2h  amount_count_2h  amount_avg_2h  amount_max_12h  \\\n",
       "1763         144.56              5.0        28.9120          161.75   \n",
       "1764          47.75              2.0        23.8750           68.16   \n",
       "1765          91.11              4.0        22.7775          121.96   \n",
       "1766          37.68              3.0        12.5600           71.63   \n",
       "1767          76.87              4.0        19.2175          159.32   \n",
       "\n",
       "      amount_sum_12h  amount_count_12h  ...  es_home_sum_14d  \\\n",
       "1763         1017.80              33.0  ...              0.0   \n",
       "1764          653.02              24.0  ...              0.0   \n",
       "1765         1001.32              32.0  ...              2.0   \n",
       "1766         1052.44              37.0  ...              0.0   \n",
       "1767         1189.73              39.0  ...              0.0   \n",
       "\n",
       "      es_travel_sum_14d  es_leisure_sum_14d  gender_F  gender_M   step  \\\n",
       "1763                1.0                 0.0       1.0       0.0   96.0   \n",
       "1764                0.0                 0.0       0.0       1.0  134.0   \n",
       "1765                0.0                 0.0       1.0       0.0  141.0   \n",
       "1766                0.0                 0.0       0.0       1.0  101.0   \n",
       "1767                0.0                 0.0       0.0       1.0   40.0   \n",
       "\n",
       "      amount  timestamp_hour  timestamp_day_of_week  label  \n",
       "1763   24.02            14.0                    2.0    0.0  \n",
       "1764   26.81            14.0                    2.0    0.0  \n",
       "1765   14.95            14.0                    2.0    0.0  \n",
       "1766   13.62            14.0                    2.0    0.0  \n",
       "1767   12.82            14.0                    2.0    0.0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview your dataset\n",
    "train_dataset.to_dataframe().tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Train models and choose the highest accuracy\n",
    "\n",
    "With MLRun, you can easily train different models and compare the results. In the code below, you train three different models.\n",
    "Each one uses a different algorithm (random forest, XGBoost, adabost), and you choose the model with the highest accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Sklearn classifier function from the functions hub\n",
    "classifier_fn = mlrun.import_function(\"hub://auto_trainer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:43:23,870 [info] starting run training uid=946725e1c01f4e0ba9d7eb62f7f24142 DB=http://mlrun-api:8080\n",
      "> 2023-02-15 14:43:24,069 [info] Job is running in the background, pod: training-68dct\n",
      "> 2023-02-15 14:43:58,472 [info] test_set or train_test_split_size are not provided, setting train_test_split_size to 0.2\n",
      "> 2023-02-15 14:44:00,031 [info] label columns: label\n",
      "> 2023-02-15 14:44:00,031 [info] Sample set not given, using the whole training set as the sample set\n",
      "> 2023-02-15 14:44:00,278 [info] training 'transaction_fraud_rf'\n",
      "/usr/local/lib/python3.9/site-packages/sklearn/calibration.py:1000: FutureWarning:\n",
      "\n",
      "The normalize argument is deprecated in v1.1 and will be removed in v1.3. Explicitly normalizing y_prob will reproduce this behavior, but it is recommended that a proper probability is used (i.e. a classifier's `predict_proba` positive class or `decision_function` output calibrated with `CalibratedClassifierCV`).\n",
      "\n",
      "> 2023-02-15 14:44:03,298 [info] test_set or train_test_split_size are not provided, setting train_test_split_size to 0.2\n",
      "> 2023-02-15 14:44:04,277 [info] label columns: label\n",
      "> 2023-02-15 14:44:04,277 [info] Sample set not given, using the whole training set as the sample set\n",
      "> 2023-02-15 14:44:04,281 [info] training 'transaction_fraud_xgboost'\n",
      "/usr/local/lib/python3.9/site-packages/sklearn/calibration.py:1000: FutureWarning:\n",
      "\n",
      "The normalize argument is deprecated in v1.1 and will be removed in v1.3. Explicitly normalizing y_prob will reproduce this behavior, but it is recommended that a proper probability is used (i.e. a classifier's `predict_proba` positive class or `decision_function` output calibrated with `CalibratedClassifierCV`).\n",
      "\n",
      "> 2023-02-15 14:44:07,773 [info] test_set or train_test_split_size are not provided, setting train_test_split_size to 0.2\n",
      "> 2023-02-15 14:44:09,037 [info] label columns: label\n",
      "> 2023-02-15 14:44:09,037 [info] Sample set not given, using the whole training set as the sample set\n",
      "> 2023-02-15 14:44:09,040 [info] training 'transaction_fraud_adaboost'\n",
      "/usr/local/lib/python3.9/site-packages/sklearn/calibration.py:1000: FutureWarning:\n",
      "\n",
      "The normalize argument is deprecated in v1.1 and will be removed in v1.3. Explicitly normalizing y_prob will reproduce this behavior, but it is recommended that a proper probability is used (i.e. a classifier's `predict_proba` positive class or `decision_function` output calibrated with `CalibratedClassifierCV`).\n",
      "\n",
      "> 2023-02-15 14:44:11,957 [info] best iteration=1, used criteria max.accuracy\n",
      "> 2023-02-15 14:44:12,668 [info] To track results use the CLI: {'info_cmd': 'mlrun get run 946725e1c01f4e0ba9d7eb62f7f24142 -p fraud-demo-dani', 'logs_cmd': 'mlrun logs 946725e1c01f4e0ba9d7eb62f7f24142 -p fraud-demo-dani'}\n",
      "> 2023-02-15 14:44:12,668 [info] Or click for UI: {'ui_url': 'https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/946725e1c01f4e0ba9d7eb62f7f24142/overview'}\n",
      "> 2023-02-15 14:44:12,669 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>fraud-demo-dani</td>\n",
       "      <td><div title=\"946725e1c01f4e0ba9d7eb62f7f24142\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/946725e1c01f4e0ba9d7eb62f7f24142/overview\" target=\"_blank\" >...f7f24142</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Feb 15 14:43:57</td>\n",
       "      <td>completed</td>\n",
       "      <td>training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=dani</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=dani</div><div class=\"dictlist\">mlrun/client_version=1.3.0-rc23</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div></td>\n",
       "      <td><div title=\"store://feature-vectors/fraud-demo-dani/transactions-fraud\">dataset</div></td>\n",
       "      <td><div class=\"dictlist\">label_columns=label</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=1</div><div class=\"dictlist\">accuracy=1.0</div><div class=\"dictlist\">f1_score=1.0</div><div class=\"dictlist\">precision_score=1.0</div><div class=\"dictlist\">recall_score=1.0</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7b487c49\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/1/feature-importance.html\">feature-importance</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/training/1/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7b487c49\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/1/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7b487c49\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/1/roc-curves.html\">roc-curves</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7b487c49\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/1/calibration-curve.html\">calibration-curve</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/training/1/model/\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7b487c49\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/0/iteration_results.csv\">iteration_results</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7b487c49\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/0/parallel_coordinates.html\">parallel_coordinates</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result7b487c49-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result7b487c49-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result7b487c49\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result7b487c49-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/946725e1c01f4e0ba9d7eb62f7f24142/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:44:15,576 [info] run executed, status=completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunObject at 0x7f3288543e20>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare the parameters list for the training function\n",
    "# you use 3 different models\n",
    "training_params = {\n",
    "    \"model_name\": [\n",
    "        \"transaction_fraud_rf\",\n",
    "        \"transaction_fraud_xgboost\",\n",
    "        \"transaction_fraud_adaboost\",\n",
    "    ],\n",
    "    \"model_class\": [\n",
    "        \"sklearn.ensemble.RandomForestClassifier\",\n",
    "        \"sklearn.ensemble.GradientBoostingClassifier\",\n",
    "        \"sklearn.ensemble.AdaBoostClassifier\",\n",
    "    ],\n",
    "}\n",
    "\n",
    "# Define the training task, including your feature vector, label and hyperparams definitions\n",
    "train_task = mlrun.new_task(\n",
    "    \"training\",\n",
    "    inputs={\"dataset\": transactions_fv.uri},\n",
    "    params={\"label_columns\": \"label\"},\n",
    ")\n",
    "\n",
    "train_task.with_hyper_params(training_params, strategy=\"list\", selector=\"max.accuracy\")\n",
    "\n",
    "# Specify your cluster image\n",
    "classifier_fn.spec.image = \"mlrun/mlrun\"\n",
    "\n",
    "# Run training\n",
    "classifier_fn.run(train_task, local=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Perform feature selection\n",
    "\n",
    "As part of the data science process, try to reduce the training dataset's size to get rid of bad or unuseful features and save computation time.\n",
    "\n",
    "Use your ready-made feature selection function from MLRun's [`hub://feature_selection`](https://github.com/mlrun/functions/blob/development/feature_selection/feature_selection.ipynb) to select the best features to keep on a sample from your dataset, and run the function on that.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:44:16,098 [info] starting run feature_extraction uid=da55327c222f4a9389232f25fc6b9739 DB=http://mlrun-api:8080\n",
      "> 2023-02-15 14:44:16,262 [info] Job is running in the background, pod: feature-extraction-pv66m\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>fraud-demo-dani</td>\n",
       "      <td><div title=\"da55327c222f4a9389232f25fc6b9739\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/da55327c222f4a9389232f25fc6b9739/overview\" target=\"_blank\" >...fc6b9739</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Feb 15 14:45:50</td>\n",
       "      <td>completed</td>\n",
       "      <td>feature_extraction</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=dani</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=dani</div><div class=\"dictlist\">mlrun/client_version=1.3.0-rc23</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div><div class=\"dictlist\">host=feature-extraction-pv66m</div></td>\n",
       "      <td><div title=\"store://feature-vectors/fraud-demo-dani/transactions-fraud\">df_artifact</div></td>\n",
       "      <td><div class=\"dictlist\">k=18</div><div class=\"dictlist\">min_votes=2</div><div class=\"dictlist\">label_column=label</div><div class=\"dictlist\">output_vector_name=transactions-fraud-short</div><div class=\"dictlist\">ignore_type_errors=True</div></td>\n",
       "      <td><div class=\"dictlist\">top_features_vector=store://feature-vectors/fraud-demo-dani/transactions-fraud-short</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/f_classif.html\">f_classif</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/mutual_info_classif.html\">mutual_info_classif</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/chi2.html\">chi2</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/f_regression.html\">f_regression</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/LinearSVC.html\">LinearSVC</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/LogisticRegression.html\">LogisticRegression</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result3d149a46\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/feature_extraction/0/ExtraTreesClassifier.html\">ExtraTreesClassifier</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/feature_extraction/0/feature_scores.parquet\">feature_scores</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/feature_extraction/0/max_scaled_scores_feature_scores.parquet\">max_scaled_scores_feature_scores</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/feature_extraction/0/selected_features_count.parquet\">selected_features_count</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/feature_extraction/0/selected_features.parquet\">selected_features</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result3d149a46-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result3d149a46-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result3d149a46\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result3d149a46-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/da55327c222f4a9389232f25fc6b9739/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:46:05,989 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "feature_selection_fn = mlrun.import_function(\"hub://feature_selection\")\n",
    "\n",
    "feature_selection_run = feature_selection_fn.run(\n",
    "    params={\n",
    "        \"k\": 18,\n",
    "        \"min_votes\": 2,\n",
    "        \"label_column\": \"label\",\n",
    "        \"output_vector_name\": fv_name + \"-short\",\n",
    "        \"ignore_type_errors\": True,\n",
    "    },\n",
    "    inputs={\"df_artifact\": transactions_fv.uri},\n",
    "    name=\"feature_extraction\",\n",
    "    handler=\"feature_selection\",\n",
    "    local=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>amount_max_2h</th>\n",
       "      <th>amount_sum_2h</th>\n",
       "      <th>amount_count_2h</th>\n",
       "      <th>amount_avg_2h</th>\n",
       "      <th>amount_max_12h</th>\n",
       "      <th>amount_sum_12h</th>\n",
       "      <th>amount_count_12h</th>\n",
       "      <th>amount_avg_12h</th>\n",
       "      <th>amount_max_24h</th>\n",
       "      <th>amount_sum_24h</th>\n",
       "      <th>amount_count_24h</th>\n",
       "      <th>amount_avg_24h</th>\n",
       "      <th>es_transportation_sum_14d</th>\n",
       "      <th>es_health_sum_14d</th>\n",
       "      <th>es_otherservices_sum_14d</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>54.55</td>\n",
       "      <td>118.62</td>\n",
       "      <td>4.0</td>\n",
       "      <td>29.655</td>\n",
       "      <td>70.47</td>\n",
       "      <td>805.10</td>\n",
       "      <td>27.0</td>\n",
       "      <td>29.818519</td>\n",
       "      <td>85.97</td>\n",
       "      <td>1730.23</td>\n",
       "      <td>58.0</td>\n",
       "      <td>29.831552</td>\n",
       "      <td>120.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>31.14</td>\n",
       "      <td>31.14</td>\n",
       "      <td>1.0</td>\n",
       "      <td>31.140</td>\n",
       "      <td>119.50</td>\n",
       "      <td>150.64</td>\n",
       "      <td>2.0</td>\n",
       "      <td>75.320000</td>\n",
       "      <td>119.50</td>\n",
       "      <td>330.61</td>\n",
       "      <td>5.0</td>\n",
       "      <td>66.122000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>218.48</td>\n",
       "      <td>365.30</td>\n",
       "      <td>5.0</td>\n",
       "      <td>73.060</td>\n",
       "      <td>218.48</td>\n",
       "      <td>1076.37</td>\n",
       "      <td>25.0</td>\n",
       "      <td>43.054800</td>\n",
       "      <td>218.48</td>\n",
       "      <td>1968.00</td>\n",
       "      <td>59.0</td>\n",
       "      <td>33.355932</td>\n",
       "      <td>107.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>34.93</td>\n",
       "      <td>118.22</td>\n",
       "      <td>5.0</td>\n",
       "      <td>23.644</td>\n",
       "      <td>79.16</td>\n",
       "      <td>935.26</td>\n",
       "      <td>31.0</td>\n",
       "      <td>30.169677</td>\n",
       "      <td>89.85</td>\n",
       "      <td>2062.69</td>\n",
       "      <td>68.0</td>\n",
       "      <td>30.333676</td>\n",
       "      <td>116.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>77.76</td>\n",
       "      <td>237.95</td>\n",
       "      <td>5.0</td>\n",
       "      <td>47.590</td>\n",
       "      <td>95.71</td>\n",
       "      <td>1259.07</td>\n",
       "      <td>37.0</td>\n",
       "      <td>34.028919</td>\n",
       "      <td>95.71</td>\n",
       "      <td>2451.98</td>\n",
       "      <td>72.0</td>\n",
       "      <td>34.055278</td>\n",
       "      <td>122.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      amount_max_2h  amount_sum_2h  amount_count_2h  amount_avg_2h  \\\n",
       "9995          54.55         118.62              4.0         29.655   \n",
       "9996          31.14          31.14              1.0         31.140   \n",
       "9997         218.48         365.30              5.0         73.060   \n",
       "9998          34.93         118.22              5.0         23.644   \n",
       "9999          77.76         237.95              5.0         47.590   \n",
       "\n",
       "      amount_max_12h  amount_sum_12h  amount_count_12h  amount_avg_12h  \\\n",
       "9995           70.47          805.10              27.0       29.818519   \n",
       "9996          119.50          150.64               2.0       75.320000   \n",
       "9997          218.48         1076.37              25.0       43.054800   \n",
       "9998           79.16          935.26              31.0       30.169677   \n",
       "9999           95.71         1259.07              37.0       34.028919   \n",
       "\n",
       "      amount_max_24h  amount_sum_24h  amount_count_24h  amount_avg_24h  \\\n",
       "9995           85.97         1730.23              58.0       29.831552   \n",
       "9996          119.50          330.61               5.0       66.122000   \n",
       "9997          218.48         1968.00              59.0       33.355932   \n",
       "9998           89.85         2062.69              68.0       30.333676   \n",
       "9999           95.71         2451.98              72.0       34.055278   \n",
       "\n",
       "      es_transportation_sum_14d  es_health_sum_14d  es_otherservices_sum_14d  \\\n",
       "9995                      120.0                0.0                       0.0   \n",
       "9996                        0.0                7.0                       0.0   \n",
       "9997                      107.0                5.0                       1.0   \n",
       "9998                      116.0                0.0                       0.0   \n",
       "9999                      122.0                0.0                       0.0   \n",
       "\n",
       "      label  \n",
       "9995      0  \n",
       "9996      0  \n",
       "9997      0  \n",
       "9998      0  \n",
       "9999      0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlrun.get_dataitem(feature_selection_run.outputs[\"top_features_vector\"]).as_df().tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Train your models with top features\n",
    "\n",
    "Following the feature selection, you train new models using the resultant features. You can observe that the accuracy \n",
    "and other results remain high,\n",
    "meaning you get a model that requires less features to be accurate and thus less error-prone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:46:06,131 [info] starting run training uid=4ac3afbfb6a1409daa1e834f8f153295 DB=http://mlrun-api:8080\n",
      "> 2023-02-15 14:46:07,756 [info] Job is running in the background, pod: training-hgz6t\n",
      "> 2023-02-15 14:46:17,141 [info] test_set or train_test_split_size are not provided, setting train_test_split_size to 0.2\n",
      "> 2023-02-15 14:46:17,731 [info] label columns: label\n",
      "> 2023-02-15 14:46:17,732 [info] Sample set not given, using the whole training set as the sample set\n",
      "> 2023-02-15 14:46:18,031 [info] training 'transaction_fraud_rf'\n",
      "/usr/local/lib/python3.9/site-packages/sklearn/calibration.py:1000: FutureWarning:\n",
      "\n",
      "The normalize argument is deprecated in v1.1 and will be removed in v1.3. Explicitly normalizing y_prob will reproduce this behavior, but it is recommended that a proper probability is used (i.e. a classifier's `predict_proba` positive class or `decision_function` output calibrated with `CalibratedClassifierCV`).\n",
      "\n",
      "> 2023-02-15 14:46:21,793 [info] test_set or train_test_split_size are not provided, setting train_test_split_size to 0.2\n",
      "> 2023-02-15 14:46:22,767 [info] label columns: label\n",
      "> 2023-02-15 14:46:22,767 [info] Sample set not given, using the whole training set as the sample set\n",
      "> 2023-02-15 14:46:22,770 [info] training 'transaction_fraud_xgboost'\n",
      "/usr/local/lib/python3.9/site-packages/sklearn/calibration.py:1000: FutureWarning:\n",
      "\n",
      "The normalize argument is deprecated in v1.1 and will be removed in v1.3. Explicitly normalizing y_prob will reproduce this behavior, but it is recommended that a proper probability is used (i.e. a classifier's `predict_proba` positive class or `decision_function` output calibrated with `CalibratedClassifierCV`).\n",
      "\n",
      "> 2023-02-15 14:46:28,944 [info] test_set or train_test_split_size are not provided, setting train_test_split_size to 0.2\n",
      "> 2023-02-15 14:46:29,507 [info] label columns: label\n",
      "> 2023-02-15 14:46:29,507 [info] Sample set not given, using the whole training set as the sample set\n",
      "> 2023-02-15 14:46:29,511 [info] training 'transaction_fraud_adaboost'\n",
      "/usr/local/lib/python3.9/site-packages/sklearn/calibration.py:1000: FutureWarning:\n",
      "\n",
      "The normalize argument is deprecated in v1.1 and will be removed in v1.3. Explicitly normalizing y_prob will reproduce this behavior, but it is recommended that a proper probability is used (i.e. a classifier's `predict_proba` positive class or `decision_function` output calibrated with `CalibratedClassifierCV`).\n",
      "\n",
      "> 2023-02-15 14:46:31,968 [info] best iteration=2, used criteria max.accuracy\n",
      "> 2023-02-15 14:46:32,376 [info] To track results use the CLI: {'info_cmd': 'mlrun get run 4ac3afbfb6a1409daa1e834f8f153295 -p fraud-demo-dani', 'logs_cmd': 'mlrun logs 4ac3afbfb6a1409daa1e834f8f153295 -p fraud-demo-dani'}\n",
      "> 2023-02-15 14:46:32,376 [info] Or click for UI: {'ui_url': 'https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/4ac3afbfb6a1409daa1e834f8f153295/overview'}\n",
      "> 2023-02-15 14:46:32,377 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>fraud-demo-dani</td>\n",
       "      <td><div title=\"4ac3afbfb6a1409daa1e834f8f153295\"><a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/4ac3afbfb6a1409daa1e834f8f153295/overview\" target=\"_blank\" >...8f153295</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Feb 15 14:46:16</td>\n",
       "      <td>completed</td>\n",
       "      <td>training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=dani</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=dani</div><div class=\"dictlist\">mlrun/client_version=1.3.0-rc23</div><div class=\"dictlist\">mlrun/client_python_version=3.9.16</div></td>\n",
       "      <td><div title=\"store://feature-vectors/fraud-demo-dani/transactions-fraud-short\">dataset</div></td>\n",
       "      <td><div class=\"dictlist\">label_columns=label</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=2</div><div class=\"dictlist\">accuracy=0.992503748125937</div><div class=\"dictlist\">f1_score=0.4827586206896552</div><div class=\"dictlist\">precision_score=0.5833333333333334</div><div class=\"dictlist\">recall_score=0.4117647058823529</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultfd68290e\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/2/feature-importance.html\">feature-importance</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/training/2/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultfd68290e\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/2/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultfd68290e\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/2/roc-curves.html\">roc-curves</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultfd68290e\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/2/calibration-curve.html\">calibration-curve</div><div title=\"v3io:///projects/fraud-demo-dani/artifacts/training/2/model/\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultfd68290e\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/0/iteration_results.csv\">iteration_results</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultfd68290e\" title=\"files/v3io/projects/fraud-demo-dani/artifacts/training/0/parallel_coordinates.html\">parallel_coordinates</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultfd68290e-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultfd68290e-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultfd68290e\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultfd68290e-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.vmdev94.lab.iguazeng.com/mlprojects/fraud-demo-dani/jobs/monitor/4ac3afbfb6a1409daa1e834f8f153295/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-02-15 14:46:33,094 [info] run executed, status=completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunObject at 0x7f324160af40>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define your training task, including your feature vector, label and hyperparams definitions\n",
    "ensemble_train_task = mlrun.new_task(\n",
    "    \"training\",\n",
    "    inputs={\"dataset\": feature_selection_run.outputs[\"top_features_vector\"]},\n",
    "    params={\"label_columns\": \"label\"},\n",
    ")\n",
    "ensemble_train_task.with_hyper_params(\n",
    "    training_params, strategy=\"list\", selector=\"max.accuracy\"\n",
    ")\n",
    "\n",
    "classifier_fn.run(ensemble_train_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Done!\n",
    "\n",
    "You've completed Part 2 of the model training with the feature store.\n",
    "Proceed to [Part 3](03-deploy-serving-model.html) to learn how to deploy and monitor the model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
